EXPRESS MAIL NO.: EL64013583^S 

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE ^ 

UTILITY PATENT APPLICATION TRANSMITTAL FORM 
(only for new nonprovisional applications under 37 CFR 1.53(b) 

ASSISTANT COMMISSIONER FOR PATENTS ^ 

Washington, D.C. 2023] 

BOX: PATENT APPLICATION 

SIR: 

Transmitted herewith for filing is the patent application (including Specification, Claims, Sequence 
Listing (if applicable) and Abstract, 27 pages) of: 

Inventor(s): Francisco Hideki Imai and Roy Stephen Berns 

For : SYSTEM AND METHOD FOR SCENE IMAGE ACQUISITION AND 

SPECTRAL ESTIMATION USING A WIDE-BAND MULTI-CHANNEL IMAGE 
CAPTURi: 

**Ifa CONTINUING APPLICATION, please mark where appropriate and supply the requisite 
information below and in a preliminary amendment: 

[ ] continuation [ ] divisional [ ] Continuation-In-Part (CIP) 
of prior application Serial No. 

Prior application information: Examiner : 
Art Unit : 

Enclosed are: 

[X] 7 sheets of informal drawings. 

[ ] Signed Combined Declaration and Power of Attorney ( pages). 

[ ] Copy of signed Combined Declaration and Power of Attorney ( pages) from a prior 

application (1.63(d) (for continuation/divisional). 

[ ] Signed statement deleting inventor(s) named in prior application ( pages) (1 .63(d)(2) and 

L33(b)). 

[ ] Incorporation By Reference: The entire disclosure of the prior application, from which a copy 
of the oath or declaration is supplied herewith, is considered as being part of the disclosure of the 
enclosed applicati<3n and is hereby incorporated by reference therein. 

[ ] Assignment ( pages) of the invention to - 

[ ] Assignment Transmittal Letter. 

[ ] Certified copy of a foreign priority document. 

[ ] Associate power of attorney. 



R438277.1 



-2- 

[ ] Verified statement to establish small entity status ( pages) (newly signed or copy filed in 

prior application). 

[ ] Preliminary Amendment ( pages). 

[ ] Information Disclosure Statement, form PTO-1449 ( pages) and references. 

[X] UNSIGNED Combined Declaration and Power of Attorney (2 pages). 

[ ] Statement in Accordance with 37 CFR § 1 .821(f) and computer readable 3.5" Diskette. 

[X] A self-addressed, pi-epaid postcard acknowledging receipt. 

[ ] Other: 

The Filing fee has been calculated as shown below: 



(Col. 1) (Col. 2) SMALL ENTITY LARGE ENTITY 



FOR: 


NO. FIILED 


NO. EXTRA 




RATE 


FEE 


OR 


RATE 


FEE 


BASIC FEE 


xxxxxxxx 


XXXXXXXX 




XXXX 


$355 


OR 


XXXX 


$710 


TOTAL CLAIMS 


70 - 20 = 


50 




X 9 = 


$450 


OR 


X 18 = 


$ 


INDEP CLAIMS 


8-3 = 


5 




x40= 


$200 


OR 


X80 = 


$ 


[ ] MULTIPLE DEPENDEtTT CLAIM PRESENTED 




xl35 = 


$ 0 


OR 


x270 = 


$ 


* If the Total Claims are less than 20 and Indep. Claims 
are less than 3, enter "0" in Col. 2 


TOTAL 


$1005 


OR 


TOTAL 


$ 



[ ] Please charge my E'eposit Account No. in the amount of $ . A duplicate 

copy of this sheet is enclosed. 

[X] A check in the amount of $1,005.00 to cover the filing fee is enclosed. 

[X] Address all future communications to: 

Gunnar G. Leinberg, Esq. 
NIXON PEABODY LLP 
Clinton Square, P.O. Box 31051 
Rochester, New York 14603 



Date : D ..;^<£^Zo ,2 ^oo 




Gunnar G. Leinberg 
Registration No. 35,584 



NIXON PEABODY LLP 
Clinton Square, P.O. Box 31051 
Rochester, New York 14603 
Telephone: (716)263-1014 
Facsimile: (716) 263-1600 



R438277.1 



EXPRESS MAIL CERTIFICATE 



DOCKET NO. : 1819/100111 

APPLICANTS Francisco Hideki Imai and Roy Stephen Berns 

TITLE : SYSTEM AND METHOD FOR SCENE IMAGE ACQUISITION 

AND SPECTRAL ESTIMATION USING A WIDE-BAND 
MULTI-CHANNEL IMAGE CAPTURE 

Certificate is attached to the Utility Patent Application which includes 
Specification, Claims, and Abstract (27 pages) and UNSIGNED Combined Declaration 
and Power of Attorney (2 pages) of the above-named application. 

"EXPRESS MAIL" NUMBER: EL640135838US 
DATE OF DEPOSIT: October 20, 2000 

I hereby cesrtify that this paper or fee is being deposited with the United States 
Postal Service "Express Mail Post Office to Addressee" service under 37 CFR 1.10 on the 
date indicated above and is addressed to the Assistant Commissioner for Patents, Box Patent 
Application, Washington, D.C. 20231. 

Jo Ann Whalen 

(Typed or printed name of person mailing 
paper or fee) 

(Signatur4^f person mailing paper or fee) 



R438268.1 



TITLE: 



SYSTEM AND METHOD FOR SCENE IMAGE 
ACQUISITION AND SPECTRAL ESTIMATION USING A 
WIDE-BAND MULTI-CHANNEL IMAGE CAPTURE 



INVENTORS: FRANCISCO HIDEKI IMAI & ROY STEPHEN BERNS 



DOCKET NO.: 1819/100111 



- 1 - 

SYSTEM AND METHOD FOR SCENE IMAGE ACQUISITION AND 
SPECTRAL ESTIMATION USING A WIDE-BAND 
MULTI-CHANNEL IMAGE CAPTURE 

5 This application claims the benefit of U.S. Provisional Patent Application 

Serial No. 60/160,565 filed on October 20, 1999, which is herein incorporated by 
reference. 

FIELD OF THE INVENTION 

The invention relates generally to methods for capturing multi-spectral 
10 images and, more particularly, to a method and system for spectral image 
representation capture used for accurate spectral estimation using statistical 
analysis of the sampled spectral reflectances of objects in the scene. 

BACKGROUND OF THE INVENTION 

15 Traditional image capture involves the concatenation of two reproduction 

processes, photography and scaiming. Unfortunately, traditional photography or 
the use of digital cameras using three channels is inherently device metameric and 
cannot reproduce the colors of the original object under different illuminants and 
for different observers. As a consequence, large color distortions can result during 

20 the image recording process. In other words, the colors of reproduced images 

may not accurately match the colors of the original scene or object. Furthermore, 
the variance in match equality due to device metamerisin can be large, resulting in 
a dramatic reduction in color quality. Consequently, vmless the color components 
in a reproduce(d image spectrally match those of the original image, metamerism 

25 will occur resulting in an observable difference between the original and the 
reproduced image imder different light. 

Metamerism is a situation in which spectrally different color stimuli look 
alike to a humjui because they have the same tristimulus values. Metamerism also 
30 defines appareat changes in color in a reproduction as compared to the original, as 
seen by an observer under different types of illumination, e.g., daylight vs. 
incandescent light, or by different observers under a single illumination. The 
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reason for metamerism is that the observed color is a product of the color in the 
image and the color of the illumination, different color components in an image 
reflect the light differently. Device metamerism is a situation in which spectrally 
different color stimuli are represented with identical records by an image capture 
5 system. 

Image editing is used to correct these observable differences between the 
original and the reproduced image. Unfortunately, editing not only consumes 
time and resources, but it is limited in how far it corrects the inherent limitations 
10 in color photography. Although the images can be pleasing, they are often 

unacceptable in terms of color accuracy. As a result, it is impossible to accurately 
capture original objects using the conventional techniques of photography and 
scanning. 

15 To ensure that a color match occurs for all observers and under all lighting 

conditions a spectral match must be achieved. As a result, an image acquisition 
system should be able to provide signals that lead to a multi-spectral description 
of a scene. To obtain a multi-spectral description of a scene a variety of different 
multi-spectral imaging methods and systems have been disclosed, such as in: U.S. 

20 Patent No. 3,684,824; U.S. Patent No. 3,714,875; U.S. Patent No. 3,720,146; U.S. 
Patent No. 3,758,707; U.S. Patent No. 4,134,683; U.S. Patent No. 4,866,454; U.S. 
Patent No. 5,109,276; U.S. Patent No. 5,889,554; and U.S. Patent No. 5,900,942 
which are all herein incorporated by reference. Many of these systems rely upon 
muhi-spectral image acquisition where light reaching a monochrome digital 

25 camera is attenuated by a set of narrow-band interference filters which are shifted 
in series in front of the camera lens. 

Unfortunately, when using interference filters for image acquisition a 
major problem is caused by the transmittance characteristic of the filters which is 
30 dependent upon the angle of incidence. For example, in order to image a painting 
with horizontal dimensions of one meter from a distance of two meters between 
the painting and the filter, there is angle of incidence -14° for points in the 
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extremities. Simulations have shown that this causes color differences of 2 AE*ab 
units in relation to the image obtained at 0° angle of incidence. 

Another problem of using interference filters in image acquisition is that 
5 the surfaces of the interference filters are not exactly coplanar which results in 
spatial shifts ajid distortions within the captiired image. Further, there may be 
inter-reflections caused by light bouncing between the interference filters and the 
camera lens. As a result, these technical problems have prevented the realization 
of practical multi- spectral imaging using interference filters without a 
10 considerable degree of expertise in multi-spectral imaging as well as complex 
image processing. 

SUMMARY OF THE INVENTION 

A metfiod for multi-spectral image capture of a first scene in accordance 
1 5 with one embc'diment of the present invention includes acquiring a first series of 
images of the Jirst scene with one or more image acquisition systems and filtering 
each of the first series of images of the scene with a different non-interference 
filter from a set of non-interference filters. Each of the image acquisition systems 
has two or mo]-e color channels and each of the channels has a different spectral 
20 sensitivity. Each of the non-interference filters in the set of the non-interference 
filters has a different spectral transmittance. 

An apparatus for multi- spectral image capture of a first scene in 
accordance wi th another embodiment of the present invention includes one or 

25 more image acquisition systems and a set of non-interference filters. Each of the 
image acquisition systems has two or more color channels with each of the 
channels having a different spectral sensitivity. Each of the image acquisition 
systems also acquires a first series of images of the first scene. Each of the non- 
interference filters has a different spectral transmittance, is positioned between the 

30 scene and the image acquisition system, and filters a different image in series of 
images. 
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A method for multi-spectral image capture of a first scene in accordance 
with another embodiment of the present invention includes providing two or more 
image acquisition systems which each have at least one spectrally unique color 
channel and ac;quiring a first series of images of the first scene. Each of the 
5 images of the Jfirst series of images is acquired with a different one of the image 
acquisition systems. 



An appiaratus for multi-spectral image capture of a first scene in 
accordance with another embodiment of the present invention includes two or 
10 more image ac quisition systems. Each of the image acquisition systems has at 
least one spectrally unique color channel and each image of the first series of 
images is acquired with a different one of the image acquisition systems. 



A method for multi-spectral image capture of a first scene in accordance 
1 5 with yet another embodiment of the present invention includes acquiring a first 

series of imagt^s of the first scene with one or more image acquisition systems and 
illuminating each image of the first series of images with a different illuminant 
from a set of two or more illuminants. Each of the image acquisition systems has 
two or more color channels with each of the channels having a different spectral 
20 sensitivity. Each illuminant has a different spectral power distribution. 



An apfjaratus for multi-spectral image capture of a first scene in 
accordance with yet another embodiment of the present invention includes an 
image acquisition system and a set of two or more illuminants. Each image 
25 acquisition system has two or more color channels with each of the color channels 
having a different spectral sensitivity. Each of the illuminants has a different 
spectral power distribution and illuminates one of the images of the first scene. 



A metliod for estimating spectral reflectances in accordance with yet 
30 another embodiment of the present invention includes a few steps. Samples of 
known spectral reflectances which are representative of colorants of a first scene 
are obtained. A first multi-spectral description of the first scene from the samples 
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is acquired. A transformation which maps charmels of the first multi-spectral 
description of the first scene back to the known spectral reflectances is derived. A 
second multi-spectral description of a second scene is acquired. The 
transformation is applied to the second multi-spectral description of the second 
5 scene to generate spectral reflectances for the second scene. 

A system for estimating spectral reflectances in accordance with yet 
another embodiment of the present invention includes samples, at least one image 
acquistion system, and a spectral image processing system. The samples have 

10 known spectral reflectances which are representative of colorants of a first scene. 
The image acquisition system obtains a first multi-spectral description of the first 
scene from the samples and a second multi-spectral description of a second scene. 
The spectral image processing system derives a transformation which maps 
charmels of the first multi-spectral description of the first scene back to the known 

15 spectral reflectances and applies the transformation to the second multi-spectral 
description of the second scene to generate spectral reflectances of the second 
scene. 

The present invention provides a number of advantages, including 
20 providing accuirate spectral estimation that overcomes the problems of 

metamerism inherent to traditional trichromatic digital and chemical photography. 
With the present invention, an excellent color match can be achieved xmder all 
types of illumination for all observers. 

25 Additionally, the present invention makes multi-spectral image acquisition 

faster than prior multi-spectral image acquisition. The present invention is faster 
because it captures multiple channels of the scene each time instead of the single 
channel capture in the narrow-band image acquisition. 

30 Further, the present invention makes multi-spectral image acquisition less 

expensive than the prior multi-spectral image acquisition. When absorption filters 
are used in accordance with one embodiment of the present invention, the present 
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invention requires fewer filters than prior methods with narrow-band interference 
filters and the absorption filters are generally less expensive than interference 
filters. When imulti-illuminants are used in accordance with another embodiment 
of the present invention, the present invention can be inexpensively implemented 
5 because these lilluminants are inexpensive and readily available. 

Even flirther, the present invention provides a multi-spectral image 
capture system and method which is easy to implement. The present invention 
can be used by people without a high-degree of expertise in multi-spectral image 

1 0 acquisition by simply switching the filter in front of the camera in one 
embodiment or by changing the illumination for each image in another 
embodiment. As a result, the present invention could easily be used for a variety 
of different practical purposes, such as producing consumer catalogs with accurate 
color reproduction of goods being sold and archiving artwork in museums. 

1 5 Further, the present invention overcomes the inherent problems related to spectral 
reconstruction using narrow-band interference filters required in prior multi- 
spectral image acquisition systems and methods. 

BRIEF DESCRIPTION OF THE DRAWINGS 

20 FIG. 1 A is a block diagram of a multi-spectral acquisition and spectral 

reflectance estimation system in accordance with one embodiment of the present 
invention; 

FIG. IB is a block diagram of a multi-spectral acquisition and spectral 
reflectance estiimation system in accordance with another embodiment of the 
25 present invention; 

FIG. 2A is a diagram of a scene with objects; 
FIG. 2B is a diagram of a scene with targets; 
FIG. 3 is a flow chart of a multi-spectral acquisition and spectral 
reflectance estimation method in accordance with one embodiment of the present 
30 invention; 

FIG. 4 is a flow chart of a multi-spectral image capture module for the 
system in accordance with one embodiment of the present invention; 
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FIG. 5 is a flow chart of a system characterization module in accordance 
with one embodiment of the present invention; and 

FIG. 6 is a flow chart of a system characterization module in accordance 
with one embodiment of the present invention. 

DETAILED DESCRIPTION 

Multi-spectral acquisition and spectral reflectance estimation systems 
10(1) and 10(2) in accordance with different embodiments of the present 
invention are illustrated in FIGS. 1 A and IB. The multi-spectral acquisition and 
spectral reflectance estimation systems 10(1) and 10(2) include at least one image 
acquisition system 12(l)-12(n) and a spectral image processing system 14. The 
multi-spectral acquisition and spectral reflectance estimation systems 10(1) and 
10(2) may also include at least one non-interference filter 16(l)-16(n) and/or at 
least one illuminant 18(l)-18(n). The present invention provides a number of 
advantages including providing a system and method which gives an accurate 
spectral reflectance estimation that overcomes the problems of metamerism and 
gives a system and method for multi-spectral image acquisition that is faster, less 
expensive, and easier to use than prior systems and methods. 



20 Multi-spectral imaging is a recording of optical signals utilizing two or 

more wavelength regions of the visible or non-visible (infrared and ultraviolet) 
spectrum. A multi-spectral description of a scene 20 is a recording of optical 
signals associated with the scene 20 such that it is possible to reconstruct the 
spectral reflectance and/or radiance of the scene 20. 

25 

Referring more specifically to FIGS. 1 A and IB, in these particular 
embodiments the multi-spectral acquisition and spectral reflectance estimation 
systems 10(1) and 10(2) each include a plurality of image acquisitions systems 
12(l)-12(n), although the systems 10(1) and 10(2) each could have only one 
30 image acquisilion system 12. An image acquisition or capture system 12 is any 
device that can record optical signals in electronic, physical or chemical form, 
such as a digital camera, a trichromatic digital camera, a photographic camera 



loaded with color negative film, a photographic camera loaded with color positive 
film, a color video camera, or a multi- spectral image capture system by way of 
example only. 

Image acquisition systems 12(l)-12(n) capture images or image views 
48(l)-48(n) of a scene 20 under given conditions, such as digital counts. The 
images or image views 48(l)-48(n) are representations of the scene 20. Objects 
22(l)-22(n) oi' targets 24(l)-24(n) within the scene 20 can comprise a variety of 
different physical entities, such as human beings, other life forms, inanimate 
things and/or their backgrounds. Some examples of scenes with objects 22(1)- 
22(n) and targets 24(l)-24(n) are illustrated in FIGS. 2A and 2B. 

The multi-spectral acquisition and spectral reflectance estimation systems 
10(1)-10(2) also each include a spectral image processing system 14. The spectral 
image processing system 14 includes a processor or central processing unit 
("CPU") 26, a memory 28, and one or more input/output (I/O) devices 30 which 
are all coupled together by a bus 32, although these systems can contain multiple 
processors, memories, I/O devices, and/or other components as needed or desired. 
Since the components and general operation of processing systems are well 
known to those of ordinary skill in the art, they will not be discussed here. 

In these particular embodiments shown in FIGS. 1 A and IB, the memory 
28 is programmed with the method for multi-spectral acquisition and spectral 
reflectance estimation in accordance with the present invention and which is 
described heredn. The program or programs stored in memory 28 are executed by 
the CPU 26 to impart data retrieval, display, processing, analyzing and access 
functionality in the system 14. Typically, the memory 28 will also store other 
programs to be executed by the CPU 26. Any type of storage device that is 
coupled to the CPU 26, such as a RAM or ROM, or a disk or CD ROM drive 
which can receive, read data from, and/or write data to a portable memory device, 
such as a flopj^y disk, hard disk, or CD ROM, can be used as the memory 28. 
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The multi-spectral acquisition and spectral reflectance estimation system 
10(1) shown in the embodiment in FIG. 1 A also includes a set of non-interference 
color filters 16(l)-16(n). In this particular embodiment, each of the non- 
interference filters 16(l)-16(n) is located or positioned between the scene 20 and 
5 an aperture, opening, or other imaging surface for one of the image acquisition 
systems 12(l)--12(n). Each of the non-interference filters 16(l)-16(n) filter the 
images of the scene being captured or acquired. The non-interference filters 
16(l)-16(n), may be cormected to or spaced from each of the image acquisition 
systems 12(l) -12(n). Although a set of separate non-interference filters 16(1)- 

10 16(n) is shown, other types and numbers of non-interference filters can be used. 
For example, a single filter system, such as a single color filter wheel, could be 
used. In one p>articular embodiment, at least one of the non-interference color 
filters has a spectrally wide bandpass, typically in the range of 100 nm to 200 nm 
at half height. The spectrally wide bandpass for the non-interference color filter 

1 5 provides a better signal-noise ratio. A variety of different types of non- 
interference filters 16(l)-16(n), such as absorbance filters, writable filters and 
liquid crystal lunable filters, can be used. In particular, absorbance filters have 
been found to be useful because they are less expensive and easier to use than 
interference filters. 

20 

The multi-spectral acquisition and spectral reflectance estimation system 
10(2) shown in the embodiment in FIG. IB includes a set of illuminants 18(1)- 
18(n). In this particular embodiment, each of the illuminants 18(l)-18(n) is 
located or positioned to illuminate the scene being captured by one of the image 

25 acquisition systems 12(l)-12(n). The illuminants 18(l)-18(n), may be connected 
to or spaced ftom each of the image acquisition systems 12(l)-12(n). Although a 
set of separate illuminants 18(l)-18(n) is shown, other types and numbers of 
illuminants can be used. For example, a single illumination system which 
illuminates each of the images being captured differently, e.g at different 

30 wavelengths, (;ould also be used. 
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Although the multi-spectral acquisition and spectral reflectance estimation 
systems 10(1) and 10(2) shown in FIGS. 1 A and IB include either a set or 
plurality of non-interference color filters 16(l)-16(n) or a set of illuminants 18(1)- 
18(n), the system 10 may include both one or more non-interference filters 16(1) - 
5 16(n) and one or more or illuminants 18(l)-18(n) or neither a non-interference 
filter 16 or an illuminate 18. For example, a multi-spectral acquisition and 
spectral reflectance estimation system 10 in accordance with another embodiment 
includes two or more image acquisition systems 12 where each image acquisition 
system 12 has at least one spectrally unique color channel and each image of a 
1 0 series of imageis to be captured is acquired with a different one of the image 

acquisition systems 12, without one or more non-interference filters 16(l)-16(n) 
or one or more illuminants 18(l)-18(n). 

The spectral image processing systems 10(1) and 10(2) may be coupled to 
1 5 a variety of different types of apparatuses or devices 34 to output the multi- 
spectral image or images, such as a cathode ray tube ("CRT") to display a 
reproduction of the scene 20, a printer for a catalog or other printing application, 
or a memory storage or archive device for archiving artworks. As a result, when 
these images are reproduced, the colors in the reproduced images will appear to be 
20 the same as the; original images to all observers under all illuminations. 

Referring to FIG. 3, the method in accordance with one embodiment of the 
present invention is illustrated. As discussed earlier, this method is stored as an 
executable program in the memory 28 and is executed by the spectral image 

25 processing system 14. The method includes a multi-spectral image capture 

module 36, a system characterization module 38, and a transformation-to-spectral 
reflectance module 40. Under given conditions 42, such as imaging geometry, 
illumination and filtering combinations, the multi-spectral image capture module 
36 outputs a multi-spectral scene description 44. The system characterization 

30 module 38 under the given conditions 42 gives a mapping transformation that is 
used by the transformation-to-spectral reflectance module 40 in order to convert 
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the multi-spectxal scene description 44 to scene reflectance 46 which can then be 
stored or be us;ed to reproduced the captured image or images by devices 34. 

Referring to FIG. 4, one embodiment of the multi-spectral image capture 
5 module 36 which is executed by the spectral imaging processing system 14 is 
shown. In this; particular embodiment, each of the image acquisition systems 
12(l)-12(n) capture an image or image view of the scene 20 with the objects 
22(l)-22(n), such as the one shown in FIG. 2A, using either a different non- 
interference filter 16(l)-16(n) or a different illuminant 18(l)-18(n), although other 

10 image acquisition systems 12 and methods could be used. For example, a single 
image acquisition system 12 could be used to capture each image or image view 
48(l)-48(n) with the multiple filters 16(l)-16(n) and/or illuminants 18(l)-18(n) 
shown or with a single filter system (not shown), such as a color wheel filter with 
multiple filters or a single illumination system (not shovm) which can illuminate 

15 the image diff<;rently for each image capture. The multiple image views captured 
under a series of given conditions are combined to form a multi -spectral 
description 44 of the scene 20 or of object 22 or objects 22(l)-22(n) within the 
scene 20. Signals in the first multi-spectral description may be normalized and 
adjusted to keep a photometric linear relationship. 

20 

Referring to FIG. 5, one embodiment of the system characterization 
module 38 which is executed by the spectral image processing system 14 is 
shown. In this particular embodiment, special targets 24(l)-24(n) with known 
spectral reflectances or with spectral reflectances which can later be measured are 

25 placed in a scene 20, as shown for example in FIG. 2B. These are known as 
characterization targets which are captured by the image acquisition systems 
12(l)-12(n) under the given conditions 42 using either different non-interference 
filters 16(l)-16(n) or a different illuminants 18(l)-18(n) to create characterization 
views 50(l)-5()(n), although again other image acquisition systems 10 and method 

30 could be used such as the one described with reference to FIG. 4. The multiple 
characterization views 50(l)-50(n) captured under a series of given conditions 42 
are combined to form a characterization description 52. The characterization 
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description 52 is analyzed by the spectral image processing system 14 along with 
the known or aicquired spectral reflectance data 54 to create a mapping from the 
characterization description to spectral reflectance, known as a characteristic 
mapping 56. 

5 

Referring to FIG. 6, one embodiment of the transformation-to-spectral 
reflectance module 40 which is executed by the spectral image processing system 
14 is shown. The characteristic mapping 56 derived under a series of image 
acquisition or igiven conditions 42 as described with reference to FIGS. 3 and 5 is 

10 applied to a multispectral description of a scene 20 which was captured under the 
same series of conditions 42 as described with reference to FIGS. 3 and 4. Based 
on these, a spectral reflectance image 58 of the scene 20 accurate for those objects 
22(l)-22(n) within the scene 20, which includes the same colorants of which the 
characterization targets 24(l)-24(n) comprise, is derived. Accordingly, the 

15 present invention accurately reproduces for every pixel of the original image of 
the scene 20 with objects 22(l)-22(n) the spectral signature identical to that of the 
corresponding point in the original image of the scene 20. The spectral 
reflectance image 58 can be used in a variety of applications. For example, 
image 58 may simply be stored to archive an image, such as a work of art, or can 

20 be used for archiving images of artworks, for display on the device 34, such as 
CRT, or to accurately reproduce the image with a device 34. 

The transformation may be performed in a variety of different spaces, such 
as spectral reflectance space, absorption space, and a new optimized space. If the 
25 transformation is performed in new optimized space, the new optimized space 

may be optimized to derive multi-variate normality of samples or improve spectral 
estimation accuracy. 



Set forlh below is a more detail explanation of one example of the process 
30 used to obtain the spectral reflectance of an image 58 of a scene 20. In this 

particular embodiment, the spectral reflectance of each pixel of the images taken 
from a scene 20 composed of objects 22(l)-22(n), such as the one shown in 
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FIG. 2B, can be estimated using a priori spectral analysis with direct measurement 
and imaging of the characterization targets 24(l)-24(n) to establish a relationship 
between the digital covmts and spectral reflectance. In order to relate the digital 
counts to spectral reflectance, a linear method is used. The spectral radiance, S, of 
5 the illuminant in the scene 20, as well as the spectral sensitivities, D, of the image 
acquisition system 12, the transmittances, F, of the filters 16 and the spectral 
reflectance, r, of the objects 22(l)-22(n) are measured and the resulting digital 
counts of the image view, Dc, are extracted from the imaged patches. 



1 0 The multi-spectral image acquisition can be modeled using matrix- vector 

notation. Expressing the sampled illumination spectral power distribution as 



and the object spectral reflectance as r=(ri, V2, ... r,y , where the index 
1 5 indicates the set of n wavelengths over the visible range and T the transpose 

matrix, representing the transmittance characteristics of the m filters as columns of 
F 



and the spectral sensitivity of the detector as 
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then the captured image is given by Dc=(DF)^Sr, where Dc, represents the 
digital counts. 

In this particular example, the spectral reflectance is sampled in the range 
of 400 nm to 700 nm wavelength in 10 nm intervals resulting in thirty-one 
samples, although different ranges and different intervals for examples can be 
used. Accordingly, in this particular example there are thirty-one signals to 
reconstruct the spectral reflectance of the image 58. However, it is possible to 
decrease the dimensionality of the problem by performing principal component 
analysis on the spectral samples. Given a sample population of spectral 
reflectances, it is possible to identify a small set of underlying basis functions 
whose linear combinations can be used to approximate and reconstruct members 
of the populations. Then the reconstructed sample r; is given by 

ii = Oaj (4) 

where <1> = (ei ei . . .Cp) are the set of the eigenvectors (principal 
components) used for the estimation and the coefficients (eigenvalues) associated 
with the eigenvectors are ai = (ai a2 . . . apf where the index p < n, and 
where n is the number of samples used to perform a priori principal component 
analysis. When the eigenvalues are arranged in descending order the fi-action of 
variance explaiined by the first corresponding p vectors is 
p 

vp.^ (5) 

In this linear method, a set of spectral reflectances r is measured and then a 
set (|)of eigenvectors, which typically explain more than 99.9% of the original 
sample, is calculated by principal component analysis. Next, the set of 
eigenvalues, a, is calculated by a=(|)^r. The set of digital counts corresponding to 
the spectral samples can be calculated by the equation 

Dc=(DF)'^Sr. (6) 

and a relationship between digital counts and eigenvalues can be 
established by the equation 
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A = aDc^[DcD J]"^ (7) 
A set of spectral reflectances, r, of characterization targets 18(l)-18(n) are 
measured and then the corresponding set of eigenvectors, e, is calculated by 
principal comjDonent analysis. The set of eigenvalues, a, corresponding to the 
eigenvectors, e, is calculated using the spectral reflectances, r. The same set of 
patches are imaged as scenes 20 and the multi-spectral description is given of the 
captured image views 48(l)-48(n) are given by the digital counts. The 
eigenvectors corresponding to the spectral reflectances are used to derive the 
characterization mapping 56 given by the transformation matrix A. When the 
image is captured, using for example an R,G,B trichromatic camera and either a 
set of non-interference filters 16(l)-16(n) or under different illuminants 18(1)- 
18(n) as show]i in FIGS. lA and IB although other systems 10 could be used, the 
spectral reflectance 58 of each pixel of the image can be calculated using matrix A 
and equation 4. 

Accordingly, in one embodiment a direct matrix transformation from 
digital counts to the spectral reflectances is derived. An eigenvector analysis is 
also used to de;rive the transformation. Further, a Wiener estimate transformation 
from digital counts to the spectral reflectances which accounts for noise 
information may also be derived and used. 



Thus, the present invention provides a number of advantages by allowing: 

1 . The possibility of accurate spectral reflectance estimations by 
overcoming the problems of device metamerism resulting from traditional image 

25 digitization using photography and scanning; 

2. Overcoming the inherent problems related to spectral 
reconstruction using narrow-band interference filters; 

3. The implementation of a multi-spectral image acquisition system 
that is faster, l(;ss expensive, and easier to use than prior methods and systems. 



30 
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EXAIVLPLES 

By way of example only, some embodiments of the present invention were 
implemented Eind tested for various targets and are described below. In these 
examples, two different image acquisition systems 10 and three targets 18 were 
5 tested. More specifically, the image acquisition systems 10, non-interference 
filters 16, and illuminants 18 used were a high-resolution trichromatic IBM 
PRO\3000 digital camera system (3,072 x 4,096 pixels, R, G, B filter wheel, 12 
bits per chann<jl that has a 45°/0° imaging configuration using tungsten 
illumination) eind a Kodak DCS560 digital camera (3,040 x 2,008 pixels, built-in 

10 R, G, B array sensors, 12 bits per channel). Both IBM PRO\3000 and Kodak 
DCS560 digital camera systems provide linear TIFF data files. The spectral 
sensitivities of the IBM PRO\3000 digital camera system were measured, as well 
as the spectral radiant power of the illuminant used in this imaging system. A 
Gretag,Macbei:h ColorChecker and two paintings as well as their corresponding 

15 painted patches were imaged. One of the paintings and its corresponding painting 
patches were generated using a mixture of GALERIA acrylic paints produced by 
Winsor & Nevrton. The acrylic painted patches were made with mixtures of two 
and three colo]-ants generating 218 patches. The other paint and the corresponding 
painted patches were generated using post-color paints. The post-color painted 

20 patches were made with mixtures of two colorants generating 105 patches. The 
paint produced! using post-color paints, as well as its corresponding patches were 
coated with Kiylon Kamar Varnish that is a non-yellowing protection. 

Different combinations of trichromatic signals were obtained from either 
25 multi-illuminaat or multi-filter approaches. In the multi-filter approach, for both 
examples, tricliromatic signals without filtering, the trichromatic signals with a 
light-blue lilteir (Kodak Wratten filter number 38), and the trichromatic signals 
with very-light-green filter (Kodak Wratten filter number 66) positioned in front 
of an aperture to the camera lens of the image acquisition systems 12 were 
30 combined. Fo]- the multi-illuminant approach, the portability of Kodak DCS560 
digital camera, was used to combine different trichromatic signals obtained from 
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targets imaged in the GretagMacbeth SpectraLight II Booth under Illuminants A 
and filtered tungsten, simulating D65. 

In summary, in order to verify the performance of these particular 
5 examples of the present invention, two different trichromatic camera systems, one 
with R, G, B filter wheel and the other with a built-in R, G, B array, were used to 
image three dijfferent targets comprising the GretagMacbeth ColorChecker and 
two sets of painted patches (one made using acrylic paints and the other made 
using post-color paints). The multi-illuminant and multi-filter approaches were 
10 also compared for one of the camera systems. The performance for each 
approach, targ<rt and camera system was evaluated in terms of colorimetric 
accuracy, spectral reflectance factor rms error and metameric index that gives an 
indication of s]3ectral match in terms of colorimetric metric comparing a standard 
illumination condition to another designated illuminant. 

15 

The results of these experiments for these particular examples showed: 
1 . The performance of principal component analysis depends on the 

particular samples and that the use of six eigenvectors to reconstruct spectra is a 

compromise btjtween accuracy and cost; 
20 2. Different combinations of various absorption filters do not affect 

the performanc;e of the spectral reconstruction according to the spectral estimation 

using simulated digital counts from the IBM PRO\3000 digital camera system; 

3. The spectral reconstruction using a trichromatic camera with multi- 
filter approach from measured digital counts produced similar results in two 

25 different digital camera systems; 

4. The spectral estimation using trichromatic signals with multi-filter 
and multi-illuminant approaches presented similar performance; and 

5. The estimation of spectral reflectance using trichromatic digital 
camera and eitiier multi-filtering or multi-illumination presented colorimetric 

30 accuracy similar to the traditional approaches using monochrome camera and 
interference narrow-band filters (e.g. the experiments performed with a Kodak 
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DCS200 camera and seven interference filters, and the MARC camera in the 
VASARI project). 

Accordingly, the present invention overcomes some inherent problems of 
5 imaging using a traditional monochrome camera combined with interference 

filters and reduces the cost and complexity of the image acquisition system while 
preserving its (^olorimetric and spectral accuracy. Additionally, the present 
invention makiss the image acquisition easier than the traditional monochrome 
camera and interference-filter-based multi-spectral acquisition. Further, in one 

1 0 embodiment o Fthe present invention having the conventional trichromatic signal 
recorded in the: trichromatic-based multimedia imaging devices also makes it 
easier to display the image on a CRT through appropriate color management. 
Even further, the spectral images generated by the present invention are also 
fundamental tci the multi-ink printing system that can select a subset of inks that 

1 5 achieve a spectral match between original objects and their printed reproductions. 

Having thus described the basic concept of the invention, it will be rather 
apparent to those skilled in the art that the foregoing detailed disclosure is 
intended to be presented by way of example only, and is not limiting. Various 
20 alternations, improvements, and modifications will occur and are intended to those 
skilled in the art, though not expressly stated herein. These alterations, 
improvements, and modifications are intended to be suggested hereby, and are 
within the spirit and scope of the invention. Accordingly, the invention is limited 
only by the following claims and equivalents thereto 
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CLAIMS 

What is claimed is: 

1 . A method for multi-spectral image capture of a first scene, the 
method comprising: 

5 acquiring a first series of images of the first scene with one or more 

image acquisition systems, each of the image acquisition systems having two or 
more color cheinnels, each of the channels having a different spectral sensitivity; 
and 

filtering each of the first series of images of the scene with a 
10 different filter from a set of non-interference filters, each of the non-interference 
filters in the se;t of the non-interference filters having a different spectral 
transmittance. 

2. The method as set forth in claim 1 further comprising generating a 
15 multi-spectral scene description fi-om the acquired first series of filtered images. 

3. The method as set forth in claim 2 further comprising: 
acquiring a second series of images of a second scene with the one 

or more image acquisition systems; and 
20 filtering each of the second series of images of the second scene 

with a different filter from the set of color filters. 

4. The method as set forth in claim 3 further comprising generating a 
characteristic mapping from the second series of filtered images. 

25 

5. The method as set forth in claim 4 further comprising generating a 
spectral reflectance of the first scene irom the multi-spectral scene description and 
the characteristic mapping. 

30 6. The method as set forth in claim 5 further comprising using the 

generated spectral reflectance to reproduce the first scene. 
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7. The method as set forth in claim 5 further comprising storing the 
generated spectral reflectance for the first scene. 

8. The method as set forth in claim 1 further comprising illuminating 
5 each image of the first series of images with one or more illuminants. 

9. The method as set forth in claim 1 further comprising illuminating 
each image of the first series of images with an illuminant from a set of two or 
more illuminants as each of the first series of images is being acquired, each of the 

1 0 illuminants having a different spectral power distribution. 

10. The method as set forth in claim 1 wherein the filters in the set of 
color filters are non-interference filters. 

15 11. The method as set forth in claim 1 0 wherein at least one of the non- 

interference filters in the set of color filters is an absorption filter. 

12. An apparatus for multi-spectral image capture of a first scene, the 
apparatus comjprising: 
20 one or more image acquisition systems each having two or more 

color channels, each of the channels having a different spectral sensitivity, each of 
the image acquisition systems acquiring a first series of images of the first scene; 
and 

a set of non-interference filters, each of the non-interference filters 
25 in the set of the; non-interference filters has a different spectral transmittance, is 
positioned between the scene and the image acquisition system, and filters a 
different image; in series of images. 



30 



13. The apparatus as set forth in claim 12 further comprising a spectral 
image processing system which generates a multi-spectral scene description from 
the acquired fii'st series of filtered images. 
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14. The apparatus as set forth in claim 13 wherein the image 
acquisition systems acquire a second series of images of a second scene and the 
set of color filters filter each of the second series of images of the second scene 
with a different filter. 

5 

1 5. The apparatus as set forth in claim 14 wherein the spectral image 
processing system generates a characteristic mapping from the second series of 
filtered images. 

10 16. The apparatus as set forth in claim 1 5 wherein the spectral image 

processing system generates a spectral reflectance of the first scene from the 
multi- spectral scene description and the characteristic mapping. 

17. The apparatus as set forth in claim 16 further comprising a printing 
1 5 device to reproduce the first scene based on the generated spectral reflectance. 

18. The apparatus as set forth in claim 16 further comprising a memory 
device for stoning the generated spectral reflectance for the first scene. 

20 1 9. The apparatus as set forth in claim 12 further comprising one or 

more illuminants which illuminate each image of the first series of images. 

20. The apparatus as set forth in claim 12 further comprising a set of 
two or more illuminants, each of the illuminants having a different spectral power 

25 distribution anid illuminating one of the images of the first series of images. 

21 . The apparatus as set forth in claim 12 wherein the filters in the set 
of color filters are non-interference filters. 



30 22. The apparatus as set forth in claim 21 wherein at least one of the 

non-interference filters in the set of color filters is an absorption filter. 
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23. A method for multi-spectral image capture of a first scene, the 
method comprising: 

providing two or more image acquisition systems, each of the 
image acquisition system having at least one spectrally unique color channel; and 
5 acquiring a first series of images of the first scene, each of the 

images of the first series of images is acquired with a different one of the image 
acquisition systems. 

24. The method as set forth in claim 23 further comprising generating a 
10 multi- spectral scene description from the acquired first series of filtered images. 

25. The method as set forth in claim 24 further comprising acquiring a 
second series o f images of a second scene with the two or more image acquisition 
systems, each of the images of the second series of images is acquired with a 

1 5 different one of the image acquisition systems. 

26. The method as set forth in claim 25 further comprising generating a 
characteristic raapping from the second series of filtered images. 

20 27. The method as set forth in claim 26 further comprising generating a 

spectral reflectance of the first scene from the multi-spectral scene description and 
the characteristic mapping. 

28. The method as set forth in claim 27 further comprising using the 
25 generated spectral reflectance to reproduce the first scene. 

29. The method as set forth in claim 27 further comprising storing the 
generated spectral reflectance for the first scene. 
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30. An apparatus for multi-spectral image capture of a first scene, the 
apparatus comjDrising: 

two or more image acquisition systems; 
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each of the image acquisition system having at least one spectrally 
unique color channel; and 

each image of the first series of images being acquired with a 
different one of the image acquisition systems. 

5 

3 1 . The apparatus as set forth in claim 30 further comprising a spectral 
image processing system which generates a multi-spectral scene description from 
the acquired fi]-st series of filtered images. 

10 32. The apparatus as set forth in claim 31 wherein the image 

acquisition systems acquire a second series of images of a second scene. 

33. The apparatus as set forth in claim 32 wherein the spectral image 
processing system generates a characteristic mapping from the second series of 
1 5 images. 



34. The apparatus as set forth in claim 33 wherein the spectral image 
processing system generates a spectral reflectance of the first scene from the 
multi- spectral scene description and the characteristic mapping. 

35. The apparatus as set forth in claim 34 further comprising a printing 
device to reproduce the first scene based on the generated spectral reflectance. 



36. The apparatus as set forth in claim 34 further comprising a memory 
25 device for storing the generated spectral reflectance for the first scene. 



37. A method for multi-spectral image capture of a first scene, the 
method comprising: 

acquiring a first series of images of the first scene with one or more 
30 image acquisitiion systems, each of the image acquisition systems having two or 
more color channels, each of the channels having a different spectral sensitivity; 
and 
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illuminating each image of the first series of images with a 
different ilium inant from a set of two or more illuminants, each illuminant having 
a different spectral power distribution. 

5 38. The method as set forth in claim 37 further comprising generating a 

multi-spectral scene description from the acquired first series of filtered images. 

39. The method as set forth in claim 38 further comprising: 
acquiring a second series of images of a second scene with the one 

10 or more image acquisition systems; and 

illvmiinating each of the second series of images of the second 
scene differently. 

40. The method as set forth in claim 39 further comprising generating a 
15 characteristic mapping from the second series of filtered images. 

41 . The method as set forth in claim 40 further comprising generating a 
spectral reflectance of the first scene from the multi-spectral scene description and 
the characteristic mapping. 

20 

42. The method as set forth in claim 41 fiirther comprising using the 
generated spectral reflectance to reproduce the first scene. 

43. The method as set forth in claim 42 further comprising storing the 
25 generated spec;tral reflectance for the first scene. 

44. An apparatus for multi-spectral image capture of a first scene, the 
apparatus comprising: 

an image acquisition system having two or more color charmels, 
30 each of the color channels having a different spectral sensitivity; and 

a set of two or more illuminants, each illuminant having a different 
spectral power distribution and illuminating one of the images of the first scene. 
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45. The apparatus as set forth in claim 44 further comprising a spectral 
image processing system which generates a multi- spectral scene description from 
the acquired fir st series of filtered images. 

5 

46. The apparatus as set forth in claim 45 wherein the image 
acquisition systems acquire a second series of images of a second scene and the 
set of color illuiminants illuminate each of the second series of images of the 
second scene vnth a differently. 

10 

47. The apparatus as set forth in claim 46 wherein the spectral image 
processing system generates a characteristic mapping from the second series of 
illuminated images. 

15 48. The apparatus as set forth in claim 47 wherein the spectral image 

processing sys tem generates a spectral reflectance of the first scene from the 
multi-spectral scene description and the characteristic mapping. 

49. The apparatus as set forth in claim 48 further comprising a printing 
20 device to reprciduce the first scene based on the generated spectral reflectance. 

50. The apparatus as set forth in claim 48 further comprising a memory 
device for storing the generated spectral reflectance for the first scene. 

25 5 1 . A method for estimating spectral reflectances comprising: 

obtaining samples of known spectral reflectances which are 
representative of colorants of a first scene; 

acquiring a first multi-spectral description of the first scene from 

the samples; 

30 deriving a transformation which maps channels of the first multi- 

spectral descrijption of the first scene back to the known spectral reflectances; 
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acquiring a second multi-spectral description of a second scene; 

and 

applying the transformation to the second multi-spectral description of the 
second scene and generating spectral reflectances of the second scene. 

5 

52. The method as set forth in claim 51 further comprising normalizing 
and adjusting signals in the first multi-spectral description to keep a photometric 
linear relationship. 

10 53 . The method as set forth in claim 5 1 wherein the transformation is 

performed in sj^ectral reflectance space. 

54. The method as set forth in claim 5 1 wherein the transformation is 
performed in absorption space. 

15 

55. The method as set forth in claim 5 1 wherein the transformation is 
performed in a new optimized space. 

56. The method as set forth in claim 55 wherein the new optimized 
20 space is optimized to derive multi-variate normality of samples or improve 

spectral estimation accuracy. 

57. The method as set forth in claim 5 1 wherein a direct matrix 
transformation from digital counts to the spectral reflectances is derived. 

25 

58. The method as set forth in claim 5 1 wherein a Wiener estimate 
transformation from digital counts to the spectral reflectances is derived. 

59. The method as set forth in claim 58 wherein the Wiener estimate 
30 transformation accounts for noise information. 
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60. The method as set forth in claim 51 wherein an eigenvector 
analysis is used to derive the transformation. 

61 . A system for estimating spectral reflectances comprising: 

5 samples of known spectral reflectances which are representative of 

colorants of a flrst scene; 

at least one image acquisition system which obtains a first multi- 
spectral descri]3tion of the first scene irom the samples and a second multi-spectral 
description of a second scene; and 
10 a spectral image processing system that derives a transformation 

which maps ch^annels of the first multi-spectral description of the first scene back 
to the known spectral reflectances and applies the transformation to the second 
multi-spectral description of the second scene to generate spectral reflectances of 
the second scene. 

15 

62. The system as set forth in claim 61 wherein the spectral image 
processing system normalizes and adjusts signals in the first multi-spectral 
description to keep a photometric linear relationship. 

20 63. The system as set forth in claim 61 wherein the transformation is 

performed in spectral reflectance space. 

64. The system as set forth in claim 61 wherein the transformation is 
performed in cibsorption space. 

25 

65. The system as set forth in claim 61 wherein the transformation is 
performed in a new optimized space. 
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66. The system as set forth in claim 65 wherein the new optimized 
space is optimized to derive multi-variate normality of samples or improve 
spectral estimation accuracy. 
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67. The system as set forth in claim 61 wherein the spectral image 
processing system derives a direct matrix transformation from digital counts to the 
spectral reflectances. 

5 68. The system as set forth in claim 61 wherein the spectral image 

processing system derives a Wiener estimate transformation from digital counts to 
the spectral reflectances. 

69. The system as set forth in claim 68 wherein the Wiener estimate 
10 transformation accounts for noise information. 

70. The system as set forth in claim 61 wherein the spectral image 
processing system uses an eigenvector analysis to derive the transformation. 



15 
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ABSTRACT 

A system and method for multi-spectral image capture of a first scene 
includes acquiring a first series of images of the first scene with one or more 
image acquisition systems and filtering each of the first series of images of the 
5 scene with a different non-interference filter, illuminating each image of the first 
series of images with a different illuminant, or acquiring each of the images of the 
first series of images with a different image acquisition system. Each of the image 
acquisition systems has at least one color channel, each of the non-interference 
filters has a different spectral transmittance, and each of the illuminants has a 
10 different spech-al power distribution. 
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